Use of Plenoptic Otoscope Data for Aiding Medical Diagnosis

ABSTRACT

A plenoptic otoscope captures images used in making a medical diagnosis. For example, the plenoptic data can be processed to produce enhanced imagery of the ear interior, and this enhanced imagery can be used in making a medical diagnosis.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of U.S. patent applicationSer. No. 13/896,924, “Plenoptic Otoscope,” filed May 17, 2013; whichclaims priority under 35 U.S.C. §119(e) to U.S. Provisional PatentApplication Ser. No. 61/754,327, titled “Plenoptic Otoscope,” filed Jan.18, 2013. This application also claims priority under 35 U.S.C. §119(e)to U.S. Provisional Patent Application Ser. No. 61/946,267, “Use ofLightfield Otoscope Data for Aiding Medical Diagnosis,” filed Feb. 28,2014. The subject matter of all of the foregoing are incorporated hereinby reference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to otoscopes for imaging the interiorof human or animal ears.

2. Description of the Related Art

Imaging inside of the human or animal ear is a common task for doctors.Typically a doctor uses an otoscope to look inside the ear of thepatient. Such an exam is common procedure when trying to diagnose earinfections. Most doctors use a manual otoscope, which is simply amagnifier combined with an illuminator. The image that the doctor seesexists only in the doctor's memory. Therefore, comparing differentimages looked at different times is difficult and not objective.

There exist digital otoscopes that have a digital camera embedded in theotoscope or at the end of a fiber-optic cable that guides the light fromthe instrument head to an external module. The digital data are thenviewed on an external display. Such digital otoscopes are marketed assolutions for telemedicine applications. Cameras currently used indigital otoscope consist of conventional imaging optics and sensors.With the rapid development of mobile platforms for smart healthcareapplications, attachments for cell phones are being developed that allowthe imaging of the inside of an ear with a smartphone for illumination,image capture, and display.

The features that doctors analyze when trying to make a diagnosis forear inflammation (“otitis media”) include features such as bulging ofthe ear drum, translucency, and yellowness of tissue. However, thesefeatures are difficult to analyze from flat two-dimensional images takenby conventional cameras. Conventional otoscopes do not explicitly obtainthree-dimensional (i.e., depth) information or wavelength-dependentcharacteristics. They are limited to images of a single focal planeinside the ear canal. Moreover, often objects such as wax or hair canobstruct the view onto the tympanic membrane (TM) or other objects ofinterest and must be removed before taking a picture of the TM,requiring some extra procedures before an otoscope can be used.

Therefore, there exists a need for improved data acquisition to allowthe extraction of three dimensions and color features more reliably.

SUMMARY OF THE INVENTION

The present invention overcomes the limitations of the prior art byproviding a plenoptic otoscope. A plenoptic otoscope can be designed toprovide good quality data for feature extraction for otitis diagnosis.In one implementation, a plenoptic sensor and an optional filter moduleare combined with a conventional digital otoscope to create a plenopticotoscope. With these additions, three-dimensional (3D) shapes,translucency and/or color information can be captured.

In one embodiment, a plenoptic otoscope includes a primary imagingsystem and a plenoptic sensor. The primary imaging system ischaracterized by a pupil plane, and includes an otoscope objective andrelay optics, which cooperate to form an image of the inside of an earat an intermediate image plane. The plenoptic sensor includes amicroimaging array positioned at the intermediate image plane and asensor array positioned at a conjugate of the pupil plane.

In one implementation, a plenoptic otoscope further includes a filtermodule positioned at a pupil plane conjugate (i.e., at the pupil planeor one of its conjugates). In one approach, the filter module is locatedin a detachable tip, and is positioned at an entrance pupil of theprimary imaging system when the detachable tip is attached to theotoscope. In this way, different filter modules can be included indetachable tips, and the filter modules can be switched in and out ofthe plenoptic otoscope by switching detachable tips.

In another implementation, a plenoptic otoscope is operable in a depthimaging mode. In the depth imaging mode, a plenoptic image (alsoreferred to as plenoptic data) captured by the sensor array is processedto provide a three-dimensional depth image of an inside of an ear.Alternately or additionally, a plenoptic otoscope is operable in aspectral imaging mode. In the spectral imaging mode, plenoptic datacaptured by the sensor array is processed to provide two or moredifferent spectral images of an inside of an ear. Disparity or depthmaps can also be determined. The plenoptic otoscope may be switchablebetween the depth imaging mode and the spectral imaging mode.

Another aspect relates to the use of the data captured by the plenopticotoscope to assist in making a medical diagnosis. For example, theplenoptic data can be processed to produce enhanced imagery of the earinterior. Data based on the enhanced imagery can then be used to assista person in making a medical diagnosis. This diagnostic data could bethe enhanced imagery itself or it could involve further processing ofthe enhanced imagery. Alternately, the diagnosis can be madeautomatically by a computer system, for example by a classifier trainedon prior data.

Enhanced imagery of the tympanic membrane is a good example. A plenopticotoscope can simultaneously capture depth and spectral information aboutthe tympanic membrane. A depth map of the tympanic membrane can produceinformation regarding its shape—whether it is bulging or retracting, andthe estimated curvature. Spectral information can include an amber oryellow image, which is especially useful to diagnose conditions of thetympanic membrane. Many diagnoses are based on shape, color and/ortranslucency, which can all be captured simultaneously by a plenopticotoscope.

Plenoptic data also includes multiple views of the same image. Thisallows the user to refocus to different depths in the image and to viewthe same image from different viewpoints. For example, the effect ofoccluding objects may be reduced by taking advantage of multiviews. Thiscould be accomplished by refocusing. Alternately, it could beaccomplished by segmenting the light field (multiple views) into depthlayers.

Examples of diagnostic data that are not images but are derived fromenhanced imagery, include classification of the tympanic membrane asbulging, retracting or neutral, estimated curvature of the tympanicmembrane, estimated color of the tympanic membrane, and features andfeature vectors reflecting any of the foregoing.

Other aspects of the invention include methods, devices, systems, andapplications related to the approaches described above and its variants.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The invention has other advantages and features which will be morereadily apparent from the following detailed description of theinvention and the appended claims, when taken in conjunction with theaccompanying drawings, in which:

FIGS. 1 a-c (prior art) are example images showing different conditionsof the ear as well as features to distinguish the conditions.

FIG. 2 is a block diagram of a plenoptic digital otoscope system.

FIG. 3 shows an optical train of a plenoptic otoscope.

FIGS. 4 a-c show optical trains of a plenoptic otoscope with filtering.

FIGS. 5 a-b show use of a filter module with different spectral filters.

FIGS. 6 a-c show additional filter modules.

FIGS. 7-8 show a plenoptic otoscope system, introducing notations ofdimensions.

FIG. 9 is a flow diagram of depth estimation and three-dimensionalinformation extraction from plenoptic otoscope data.

FIG. 10 shows depth maps estimated from plenoptic data imaging an eartrainer.

FIGS. 11 a-b are graphs showing estimates of eardrum bulging/retracting,based on the depth maps of FIG. 10.

FIGS. 12 a-c illustrate estimation of a depth map for an adult TM.

FIG. 13 is a flow diagram showing a method for selected focal planerendering.

FIG. 14 shows images corresponding to the flow diagram of FIG. 13.

FIGS. 15 a-b show image rendering at a selected focal plane with a smallsynthetic aperture and with a large synthetic aperture, respectively, toremove hair occlusions.

FIG. 15 c shows a reference image captured without any hair occlusions.

FIGS. 16 a-d show spectral images captured by a plenoptic otoscope.

The figures depict embodiments of the present invention for purposes ofillustration only. One skilled in the art will readily recognize fromthe following discussion that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the invention described herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed. To facilitate understanding, identical referencenumerals have been used where possible, to designate identical elementsthat are common to the figures.

A plenoptic otoscope design can overcome the poor data quality ofcurrent otoscopes for feature extraction for otitis diagnosis. In oneimplementation, a plenoptic sensor is added to a conventional digitalotoscope as well as an optional filter module inside the otoscopicinstrument. With these additions, three-dimensional (3D) shapes,translucency, and/or detailed color information can be captured. Thisdata and possibly other data captured by a plenoptic otoscope can beused to aid in medical diagnosis of a patient's ear.

FIGS. 1 a-c (prior art) are example images showing different conditionsof the ear as well as features to distinguish the conditions. The threeconditions shown are acute otitis media (AOM) in FIG. 1 a, otitis mediawith effusion (OME) in FIG. 1 b, and otitis media with no effusion (NOE)in FIG. 1 c. Table 1 lists some features distinguishing the conditions.More specifically, Table 1 lists otoscopic findings on tympanic membrane(TM) images associated with the above three conditions.

TABLE 1 Otoscopic findings associated with clinical diagnosticcategories on TM images AOM OME NOE Color White, pale yellow, White,amber, gray, Gray, pink markedly red blue Position Distinctly full,Neutral, retracted Neutral, retracted bulging Translucency OpacifiedOpacified, semi- Translucent opacified

As can be seen from FIGS. 1 a-c and Table 1, the three conditions of theear are different and they can be distinguished from one another basedon one or more of the following features: color, position (e.g., 3Dshape), and translucency. In order to make correct diagnosis of the earcondition, otoscopic images capturing accurate information about color,3D shape and translucency of an inside of an ear (e.g., a tympanicmembrane in an ear canal) are desirable.

FIG. 2 is a block diagram of a plenoptic digital otoscope system. Thesystem includes an otoscope objective 210, imaging optics (relay optics)220, a plenoptic sensor 230 and image processing 240. The otoscopeobjective 210 can be an imaging objective, as used in conventionalotoscopes. The imaging optics 220 works in conjunction with the otoscopeobjective 210 to form a conventional image within the otoscopeinstrument. Rather than a conventional sensor array capturing thisimage, a plenoptic sensor 230 captures the image. The plenoptic sensor230 is a sensor array with a microimaging array (e.g., a microlens arrayor pinhole array) mounted in front of it. In addition, a filter module(not shown in FIG. 2) can be inserted at a pupil plane of the opticaltrain (or at one of its conjugates) to allow spectral or other filteringof the light. The digital information extracted by the plenoptic sensor230 is sent to a computing module 240 that performs the image processingof the plenoptic data. In this way, three-dimensional and/or spectraldata can be extracted.

The plenoptic otoscope head can be mounted on top of a handle thathouses an illumination source (e.g., portable system) or can beconnected to an illumination source (e.g., wall-mounted system). Such anillumination source may be an LED light source, a standard whiteillumination source, etc. The illumination source may have polarizationcharacteristics as well. For example, it may emit unpolarized, partiallypolarized, or completely polarized (e.g., TE, TM) light.

FIG. 3 shows an optical train of a plenoptic otoscope. The plenopticotoscope includes two parts: a primary imaging system and a plenopticsensor. The primary imaging system includes an otoscope objective 320and relay optics 330. They cooperate to form a conventional image of anobject 310 (e.g., an inside of an ear, a tympanic membrane, etc.). Aplenoptic sensor (a microlens array 340 and a sensor array 350) ispositioned so that the microlens array 340 is located in theconventional image plane, which is an intermediate image plane of theprimary imaging system. The sensor array 350 then captures lightfielddata (or plenoptic data), which will be referred to as the plenopticimage of the object 310.

In one embodiment, the plenoptic image contains depth data. A computingmodule (not shown in FIG. 3) further processes the captured plenopticimage to produce three-dimensional data. This operational mode of theplenoptic otoscope may be referred to as a depth imaging mode. Forexample, in the depth imaging mode, the plenoptic data captured by thesensor array 350 may be processed to provide a three-dimensional depthimage of an inside of an ear.

Another possible operational mode of the plenoptic otoscope is aspectral imaging mode. In the spectral imaging mode, the plenoptic imagecaptured by the sensor array 350 contains spectral information and maybe processed to provide two or more different spectral images of theobject 310. In one embodiment, spectral imaging can be enabled byplacing a filter module at a pupil plane conjugate of the plenopticotoscope, as shown in FIGS. 4 a-c. The term “pupil plane conjugate” isused to refer to any plane that is a pupil plane of the primary imagingsystem or a conjugate plane of that pupil plane. For example, the termpupil plane conjugate includes the entrance pupil plane and the exitpupil plane of the primary imaging system.

FIGS. 4 a-c show different placements of the filter module. In FIG. 4 a,the filter module 410 is located at an aperture between the relay opticsand the plenoptic sensor. In FIG. 4 b, the filter module 410 is locatedat the entrance pupil. In FIG. 4 c, the filter module 410 is located atan aperture between a pair of relay lenses. In each of the embodimentsshown in FIGS. 4 a-c, the filter module 410 is positioned at a pupilplane conjugate.

In one implementation of FIG. 4 b, the filter module is contained in adetachable tip (or ring), which is attached to the plenoptic otoscope.When the tip is attached, the filter module is positioned at theentrance pupil of the first lens group, as shown in FIG. 4 b. As in aconventional otoscope, a speculum used to enter the ear canal may beattached to the detachable tip.

In one embodiment, the plenoptic otoscope is switchable between thedepth imaging mode and the spectral imaging mode. In one approach, aclear filter is used for the depth imaging mode and one or moredifferent spectral filters are used for the spectral imaging mode. Toswitch between the two modes, the filter module 410 could include onesection that is clear and another section that contains the spectralfilters. The filter module could be translated relative to the primaryimaging system, so that the appropriate section is illuminated. Anexample of this type of filter module is shown in FIG. 5. This filtermodule could be positioned in a pupil plane conjugate and translatedwithin the pupil plane conjugate to switch between the set of spectralfilters and the clear filter. In FIG. 5, the black circle shows thecross-section of the light traveling through the otoscope.

In FIG. 5 a, the light travels through the spectral filters, which aredepicted as a red rectangle, a blue rectangle, a green rectangle, and ayellow rectangle. The portion of light that passes through a colorfilter forms an image of the object (within the plenoptic image) thathas been filtered by the corresponding color filter. As a result,spectral imaging is enabled. In this example, a plenoptic image isformed, from which four different spectral images of the object (i.e., ared image, a blue image, a green image, and a yellow image) may beextracted.

In FIG. 5 b, the filter module is translated relative to the primaryimaging system so that the light travels through the clear aperture.This might be used for depth imaging, for example. In this example, aplenoptic image is formed, from which a three-dimensional depth imagemay be extracted.

This particular filter module has RGB filters for color imaging, plus ayellow filter since yellowish or amber color of tissue is an indicator,and is only shown as an example. In one embodiment, the filter modulemay include a plurality of different spectral filters. Filters havingdifferent colors and/or layouts may also be used in the filter module.For example, see U.S. patent application Ser. No. 13/040,809, filed onMay 4, 2011, which is hereby incorporated by reference in its entirety.

Spectral imaging is useful to help distinguish different ear conditions.Some of the ear conditions are shown in FIGS. 1 a-c and also in Table 1.For example, AOM is markedly red, OME features amber, and NOE containsgray and pink. In one embodiment, the filter module includes differentfilters selected to distinguish different ear conditions. Such a filtermodule is shown in FIGS. 5 a-b, e.g., the filter module containingred-green-blue filters and a yellow filter.

FIGS. 6 a-c show some additional filter modules. In FIG. 6 a, the filtermodule has a yellow filter and a transparent area. In FIG. 6 b, thefilter module has a yellow filter, an amber filter and a transparentarea. The sensor array can also be fitted with spectral filters, such asa standard Bayer RGB pattern. Thus, when the filter modules in FIG. 6 aor 6 b are used with their transparent areas, the Bayer RGB pattern maybe used to achieve color imaging. The yellow and amber filters in FIG. 6a and FIG. 6 b can be used to extract extra color information. They canbe used together with monochromatic sensors and/or RGB sensors (e.g.,sensors fitted with a standard Bayer RGB pattern). Such extra colorinformation (e.g., yellow, amber, etc.) can be used to distinguishdifferent ear conditions.

FIG. 6 c has an array of filters. The center stripe includes threespectral filters: yellow, amber and a third spectral filter. These canbe used for spectral imaging. The top right and top left filters arepolarization filters, for example to reduce reflections. Thepolarization filters may also be useful in extracting illuminationcharacteristics (e.g., when the illumination light has a certain degreeof polarization). The middle right and middle left filters aretransparent, for example for use in extracting depth information (e.g.,depth imaging). The bottom right and bottom left filters are neutraldensity filters of different densities, for example to increase thedynamic range of the plenoptic otoscope.

FIGS. 7-8 show a plenoptic otoscope system, introducing notation thatwill be used to describe different design considerations. FIG. 7 depictsa typical embodiment of a plenoptic otoscope, which includes a primaryimaging system and a plenoptic sensor. The primary imaging systemincludes two lens groups. The first lens group is the otoscopeobjective, and the second lens group is the relay optics. The plenopticsensor includes a microimaging array 340 and a sensor array 350. In FIG.8, the microimaging array is a microlens array 340, with each microlenshaving identical optical properties, such as diameter, radius ofcurvature, material, and thickness. In one embodiment, the diameter of amicrolens is chosen to be between 10 and 100 micron.

As shown in FIG. 7, the object (e.g., a tympanic membrane) is located inan object plane. It is imaged by the first lens group onto a firstintermediate image plane (which may be referred to as the relay plane),and then imaged by the second lens group onto a second intermediateimage plane where the microlens array 340 is positioned.

In many conventional otoscopes, the magnification of the primary imagingsystem is set such that the entire tympanic membrane (TM) can be imagedonto the sensor array 350 (as seen in FIGS. 1 a-c). Suppose the sensorarray 350 has a width W and a height H, and the diameter of the TM is h,then the magnification of the primary imaging system is given byM=min(W,H)/h, where min(x,y) returns the lesser value of x and y.

The average diameter for the TM of an adult is h=7 mm. Here we defineoptical system specifications for the example of a ⅓″ sensor array withwidth W=4.6 mm and height H=3.7 mm. For this sensor array, themagnification for the primary imaging system is given by M=3.7 mm/7mm=0.53. Such a magnification is typical for a conventional otoscope. Incontrast, a microscope typically has a much larger magnification (>20),and a consumer camera imaging people or natural scenes typically has amuch smaller magnification.

The total magnification of the primary imaging system is M=M1*M2, whereM1 is the magnification of the first lens group, and M2 is themagnification of the second lens group. For illustration purposes,assume M2=1. In other approaches, M2 can be any suitable number otherthan 1. In the example where M2=1, M1=M. The working F-number, N_(w), ofthe first lens group with magnification M is defined as N_(w)=(1+M)N,where N is the F-number of the primary imaging system (i.e., N=f/D1,where D1 is the diameter of the entrance pupil of the primary imagingsystem, and f is the effective focal length of the primary imagingsystem.). In one embodiment, the primary imaging system of the plenopticotoscope is faster than F/8.

The working distance, z1, for the otoscope is the distance between theobject and the first lens group. For imaging a TM, a typical workingdistance is 27-30 mm. The bones behind the TM are located approximatelyup to a distance of 15 mm from the TM. As a result, the working distancemay vary, for example, from 27 mm up to 45 mm. For illustrationpurposes, assume the working distance z1=30 mm. The entrance pupil islocated in the narrow tip of the otoscope close to the first lens group,and is generally smaller than the tip of the otoscope. The tip of anotoscope has a typical diameter of 4-5 mm in order to fit into an earcanal. Let's assume the entrance pupil to have a diameter of 2 mm. Thenthe effective focal length of the first lens group is f=N*D1=10.4 mm.The second lens group relays the image of the first lens group onto anintermediate image plane, where the microlens array 340 is positioned.The sensor array 350 is positioned at a distance z3′ behind themicrolens array 340 to capture the plenoptic image.

In one embodiment, the object is located near the hyperfocal distance ofthe first lens group. The hyperfocal distance is a distance beyond whichall objects can be brought into an acceptable focus. Mathematically, thehyperfocal distance may be expressed as p=f²/(N c)+f, where f is theeffective focal length, N is the F-number, and c is the circle ofconfusion diameter limit. In one implementation, the numerical apertureof a microlens matches the image-side numerical aperture of the primaryimaging system. That means the working F-number of the primary imagingsystem matches the F-number of the microlens. Furthermore, the distancez3′ is chosen to be equal to the focal length of the microlens. In thisconfiguration, the depth of field is bounded only in one direction, andtherefore may be particularly suitable for imaging distant objects.

In one embodiment, the object is placed at a distance z1 away from theentrance pupil of the first lens group. The distance z2 between the exitpupil of the first lens group and the relay plane is determined by thelens equation as: z2=1/(1/f1−1/z1), where f1 is the effective focallength of the first lens group.

The relationship between the first lens group and the second lens groupis given by D1_(exit)/D1′=z2/z1′, where D1_(exit) is the diameter of theexit pupil of the first lens group, D1′ is the diameter of the entrancepupil of the second lens group, and z1′ is the distance between therelay plane and the entrance pupil of the second lens group.

The distance z2′ between the exit pupil of the second lens group and theintermediate image plane is determined by the lens equation as:z2′=1/(1/f1′−1/z1′), where f1′ is the effective focal length of thesecond lens group.

The distance z3′ between the microlens array and the sensor array ischosen such that z3′=z2′M_(microlens). Here M_(microlens)=D2/D1′ exit isthe magnification of the microlens sub-system, where D2 is the diameterof the microlens (as shown in FIG. 8) and D1′_(exit) is the diameter ofthe exit pupil of the second lens group. This configuration is specificto imaging with a depth of field bounded in both directions, where theobject may not be located near the hyperfocal distance. In comparison, amicroscope typically has a much larger magnification (>20), a largerF-number (>15), and a much smaller working distance (a few millimeters).

In one embodiment, the filter module 410 is inserted at the aperture ofthe second lens group, as depicted in FIG. 7. The filter module 410 isadjustable in such a way that it can be translated laterally in the x-yplane, which is perpendicular to the optical axis (z axis) of the secondlens group. For clarity, the coordinate system is also shown in FIG. 7.Furthermore, the second lens group may have a diaphragm/iris/shutterattached to the front/back of the filter module 410. This configurationmay permit adjustment of the aperture diameter by opening and closingthe diaphragm/iris/shutter.

Switching between depth imaging mode and spectral imaging mode may beaccompanied by a change in the depth of field for the primary imagingsystem (in addition to changing filters). One way to change the depth offield is by adjusting the aperture size. For example, a larger apertureresults in a shorter depth of field, which may benefit depth imaging dueto the finer depth resolution. On the other hand, a smaller apertureresults in a longer depth of field, which may be unsuitable for depthimaging but appropriate for spectral imaging.

In one embodiment, switching between depth and spectral imaging includesopening and closing the diaphragm/iris/shutter at the aperture plane ofthe second lens group. Two example configurations are given below. Inthe first configuration, with the effective focal length f=10 mm and acircle of confusion diameter of 0.019 mm, the aperture is wide open toenable a small F-number (e.g., F/5) and a small depth of field (<2 mm).This configuration is suitable for depth imaging or perhaps for combineddepth+spectral imaging. In the second configuration, with the effectivefocal length f=10 mm and a circle of confusion diameter of 0.019 mm, theaperture is stopped down to enable a large F-number (e.g., F/16) and alarge depth of field (>3.5 mm). This configuration may be suitable forspectral imaging only.

Switching between depth imaging mode and spectral imaging mode may alsobe accompanied by a change in focus for the primary imaging system. Thismay be done via a focusing mechanism. Such a focusing mechanism (e.g., afocusing ring) may move lenses in the primary imaging system and/or movethe plenoptic sensor, so that objects at various distances can befocused onto the microlens array plane (i.e., the intermediate imageplane). In one approach, the focusing mechanism is adjusted such that aregion between 4-5 mm in front of the TM and up to 15 mm behind the TMcan be imaged in focus onto the microlens array plane. This may enabledifferent combinations of spectral and/or depth imaging at differentregions of interest. For example, it may be desirable to have both depthand spectral imaging for a region near the TM (e.g., to fullydistinguish the different ear conditions), while spectral imaging may beenough for other regions. By adjusting the focus, it is possible toselect which portion of the ear canal should “receive more attention.”For instance, one can adjust the focus with a fine step size (i.e., afine depth resolution) near the TM to increase the 3D depth informationfor that region of interest, and adjust the focus with a coarse stepsize for other regions of the ear canal.

In one embodiment, the plenoptic otoscope is in the spectral imagingmode when the primary imaging system has a depth of field >5 mm. This isuseful, for example, for imaging both the TM and the bones behind the TMin focus onto the microlens array plane. Conversely, the plenopticotoscope is in the depth imaging mode when the primary imaging systemhas a depth of field <5 mm. In this mode, depth estimation of the TM ispossible, for example, by focusing on the bones behind the TM and/or thenarrow part of the ear canal in front of the TM. Illustratively, thefirst lens group may have a working distance up to 45 mm (about 15 mmbehind the TM).

In a plenoptic otoscope, it is also possible to include a view finder toenable the examiner to view an image through the view finder of theotoscope at the time of image capture. A beam splitter or a single lensreflex can be used to split the optical path and direct the image to theplenoptic sensor and to the view finder. For example, either a singlelens reflex or a beam splitter may be inserted at the relay planebetween the first lens group and the second lens group of an otoscope(as shown in FIG. 7) to allow a medical expert to look at an ear drum,while the plenoptic image of the ear drum is captured on the sensorarray of the same otoscope.

In other embodiments, a plenoptic otoscope system may include a set ofdetachable tips. Each detachable tip includes a different filter module.Each filter module may be used for a different purpose. For example, onefilter module may be used for spectral imaging, while another filtermodule may be used for depth imaging. These detachable tips can beexchanged with one another, and are also referred to interchangeabletips. When a detachable tip is attached to the otoscope, the filtermodule included in that detachable tip is positioned at the entrancepupil of the primary imaging system.

The plenoptic otoscopes described can be designed and manufactured asoriginal plenoptic instruments. Alternately, existing otoscopes can bemodified to become plenoptic. In one embodiment, an after-marketplenoptic conversion kit may be used to convert a conventional digitalotoscope to a plenoptic digital otoscope. The conversion kit includes aplenoptic sensor with a microimaging array and a sensor array. Thedigital otoscope is equipped with a conventional sensor. During theconversion, the plenoptic sensor replaces the conventional sensor, suchthat the microimaging array (e.g., a microlens array or a pinhole array)is positioned at an image plane of the digital otoscope. For example,the microimaging array may be positioned at the plane where theconventional sensor was previously located.

FIGS. 9-16 provide additional description about the operation of aplenoptic otoscope and the use of plenoptic data (i.e., plenopticimages) for medical diagnosis. Plenoptic data of the ear canal can beprocessed to extract information about objects in the ear canal,especially three-dimensional and spectral information. Examples includethe following:

-   -   Depth map estimation of the ear canal, including the TM    -   Depth map processing to extract three-dimensional information of        the ear canal including the TM    -   TM shape estimation and classification    -   Three-dimensional ear canal segmentation, occlusion detection,        object ordering, feature extraction and feature processing for        classification of (medical) conditions of the ear    -   Three-dimensional ear-canal and TM visualization    -   Spectral measurement    -   Opacity/translucency        Further aspects include use of plenoptic data to extract object        information, optionally including displaying results of object        information to a user, such as:    -   Shape of the TM    -   Obstruction by other objects    -   Selection of object obstructed by another object    -   Multi-view/multi-focal rendering of ear canal    -   Image rendering with different synthetic aperture size    -   Removal of occlusions in the view of the ear canal

Data acquired with a plenoptic otoscope can contain volumetric data ofthe ear canal and can also provide spectral measurement as well aspolarization states of objects. Enhanced imagery such as a depth map,disparity map, spectral images, polarization images and images showingthe translucency of objects can be computed from the plenoptic data. Inaddition, higher-level information such as deformation of the shape ofthe TM, focusing on a selected object, and segmentation of the objectsin the ear canal with respect to depth in the ear canal can also beperformed. Results may be displayed to the user.

Consider first depth estimation. Given the design of a plenopticotoscope, the data obtained with that system during a single dataacquisition step can be processed to provide enhanced imagery with depthmeasurements of the TM as well as the ear canal at sub-millimeterresolution. These measurements can be used to aid the assessment of amedical condition.

FIG. 9 is a flow diagram of depth estimation and three-dimensionalinformation extraction from plenoptic data. A plenoptic otoscopeacquires 910 a lightfield, which is a set of pixels that can bere-arranged into a set of images of the ear canal, where each imagecaptures the ear canal from a different viewpoint (different viewingangle). These images, sometimes called multiviews, carry informationabout the three-dimensional shape of the ear canal and the objectsinside it. However, this information is not captured explicitly, but isestimated from the multiviews using depth estimation techniques.

An example of a method to estimate depth imagery from a lightfield isdescribed in U.S. patent application Ser. No. 14/064,090, “Processing oflight fields by transforming to scale and depth space,” which isincorporated by reference in its entirety herein. It is a multi-scaledepth estimation approach, which analyzes the 4D lightfield data and canfind a dense disparity map with sub-pixel precision. The method is basedon extrema localization in light field scale and depth spaces, which areconstructed by convolving a two-dimensional spatio-angular slice of agiven 4D lightfield with a kernel designed to represent the structure oflightfields and to provide a simple way of estimating disparity valuesfor the imaged object. Disparity values can then be converted to depthusing a mapping based on system modeling.

Other methods for depth estimation can also be used. For example,multi-view stereo depth estimation algorithms can be applied. Theseinclude algorithms that pose the depth estimation problem as an energyminimization problem, where the energy includes a data fidelity term anda depth map smoothness term. The energy function can then be optimizedusing methods such as graph cuts, belief propagation, total variation,semi-global matching, etc. Sometimes image segmentation can be used incombination to depth estimation, in order to improve the final depthaccuracy. Another approach that can be applied exploits the structure ofthe lightfield to obtain dense depth maps. An example of such a methodis an algorithm that computes the structure tensor of the light fieldslices and uses that as a data fidelity term, while using totalvariation as a smoothness term. Computationally efficient methods basedon normalized cross-correlation can also be applied to obtain a coarsedepth map.

Yet another approach is described in U.S. patent application Ser. No.14/312,586, “Disparity estimation for multiview imaging systems,” whichis incorporated by reference in its entirety herein. This estimates adepth/disparity map using multiple multiview images and taking advantageof the relationship between disparities for images taken from differentviewpoints.

Depth estimation techniques are used to obtain 920 enhanced imagery (inthis case, depth/disparity maps) of the ear canal and/or objects insideit from lightfields obtained with a plenoptic otoscope. Moreover, depthmap information can also be used to extract other diagnostic data, suchas relevant three-dimensional shape information about the ear canal, eardrum or other objects in the ear canal, in order to help inthree-dimensional visualization and diagnosis of medical conditions ofthe ear (e.g., variants of otitis media). The depth measurementspreferably are taken with respect to the front of the camera and areavailable for different spatial locations in the scene. They can be alsocalculated for objects in the field of view of the camera (e.g., theeardrum or the mallus).

Depth map processing 930 includes different methods for extraction ofrelevant three-dimensional diagnostic data of the ear canal, ear drumand/or other objects in the ear canal. For example, the curvature of theear drum can be estimated from the depth map data, by fittingone-dimensional or two-dimensional polynomials to the depth map values.Using the curvature estimate we can classify the shape of the eardruminto bulging, neutral or retracting (convex, planar or concave).Moreover, we can evaluate the amount of bulging or retracting of the eardrum. This can be used to aid in medical diagnosis. For example, see the“Position” row of Table 1 above.

FIG. 10 shows estimated depth maps obtained from plenoptic lightfieldimages of an ear trainer, where bulging/retracting was simulated byinjecting liquid into a membrane mounted on an eardrum cartridge. Thedifferent depth maps vary from retracting 2.0 mm, through neutral, tobulging 2.5 mm. In FIG. 10, the hotter colors (e.g., red) indicate thatthe eardrum is at a larger depth (i.e., farther away). The cooler colors(e.g., blue) indicate that the eardrum is at a smaller depth (i.e.,closer).

FIGS. 11 a-b are graphs showing estimates of eardrum bulging/retracting,based on the depth maps of FIG. 10. FIG. 11 a is a bar graph showing theestimating amount of bulging/retracting compared to the measured amountof bulging/retracting. The bar graph has pairs of bars for each eardrumposition, where the right bar 1110 is the estimated amount based on thedepth map and the left bar 1112 is the “ground truth” based on a directmeasurement using a micrometer. The root mean square error is less than0.3 mm. Depth resolution of less than 1 mm is achieved. Lightfields fromthe plenoptic otoscope can provide three-dimensional information of theeardrum with sub-millimeter precision.

In FIG. 11 b, a “bulging index” is based on the estimated curvature ofthe eardrum. The curvature is estimated by fitting a second degreepolynomial to one-dimensional scan lines through the depth map and thenselecting the scan line with maximum curvature. In this example, aclassifier is used to separate the data into three classes: bulgingversus normal versus retracting. A bulging index of −1 means the eardrumis retracting, +1 means the eardrum is bulging, and 0 means the eardrumis neutral. The right bar graph 1120 in each pair is the estimated indexand the left bar graph 1122 in each pair is the measured index. Theestimated index is 100% accurate. Other classification algorithms may beused as well.

The depth measurements and classification results can be used to assista human to make a diagnosis. Alternately, it may be used, possibly incombination with other data, to make an automated diagnosis.

FIGS. 10-11 show one example of depth map processing. Other types ofdepth map processing 930 include object segmentation, occlusiondetection and estimation of object ordering, three-dimensionalkeypoint/feature extraction, three-dimensional feature description,classification based on three-dimensional feature descriptions,three-dimensional object measurements, comparisons with previous depthmaps to obtain treatment progress evaluation, etc.

FIGS. 12 a-b illustrate estimation of a depth map for an adult TM. FIG.12 a is an image showing the center view extracted from the lightfielddata. It shows the TM and the malleolus (bone in the middle ear). FIG.12 b shows a depth map obtained from the plenoptic data using the methoddescribed above. The darker pixels denote closer distances (smallerdepth) and the lighter pixels denote farther distances (larger depth).FIG. 12 c is a three-dimensional rendering of the depth map of FIG. 12b. The shape of the TM is retracted in this example.

Returning to FIG. 9, another aspect of three-dimensional imaging of theear canal is the visualization of the three-dimensional information.This can be achieved, for example, by visualization of the estimateddepth maps as images (with hotter-colder colors such as in FIG. 10).This is referred to as depth map visualization, as indicated in block940 of FIG. 9. Another way to visualize the three-dimensionalinformation of the ear canal is to map the color values to the depthpoint cloud obtained from the depth map and then do three-dimensionalrendering 950 using a graphics engines. The result is athree-dimensional surface with color information that the doctor canrotate using the mouse or other computer interface. This is referred toas three-dimensional visualization 960.

Plenoptic data can also be used to compute a two-dimensional renderingof a selected focal plane, or a three-dimensional volumetric renderingof the ear canal taking advantage of multiple viewpoints. With suchvisualization, the medical professional can switch between differentviews of e.g. some hair or wax in the ear canal and the ear drum, or canswitch between views of the ear drum from different viewpoints.

The problem of “seeing through” occlusions in the ear canal can beaddressed by refocusing the lightfield data at a selected focal plane,and then rendering the image with a large synthetic aperture. The focalplane can be selected by the user in various ways. For example:

-   -   Selecting the focal plane from a depth map (such as shown in        FIG. 10) extracted from the lightfield data. The user might        click on a point in the depth map to select the focal plane, or        might click on a legend for the depth map to select the focal        plane.    -   Selecting the focal plane based on an object in a        three-dimensional visualization of the lightfield data. The user        might select an object, which in turn defines the desired focal        plane.    -   Conversely, the focal plane may be selected based on a depth        plane or object (e.g., an occluding object) to be removed from        the view.        The focal plane can also be calculated via an algorithm using        prior knowledge of object locations in the ear canal, e.g.        typical distance between otoscope speculum and TM. Once the        focal plane of interest is selected, whether by the user or        determined automatically, an algorithm can be used to refocus        extracted multiview images at the desired focal plane and can        render the output image with different aperture size.

FIG. 13 is a flow diagram showing a method for selected focal planerendering. This method is illustrated using an example simulation shownin FIG. 14. This example renders the image at the focal plane of the TMwhile removing obstructing hair in front of the TM. A plenoptic otoscopeacquires 1310 plenoptic data of the ear canal. Image 1410 shows the rawsensor data. This data includes multiple views, but the views areinterleaved with each other. Multiple views of the ear canal areextracted 1320 from the lightfield data. FIG. 14 shows the multipleviews 1420 of the ear canal. This looks like an array of identicalimages, but the images are not identical. Each image is taken from adifferent view. The location of the image within the array indicates theviewpoint from which the image was taken.

A disparity map is calculated 1330 from the different views. In thisexample, not all the views are used to calculate the disparity map.Rather, the disparity map is calculated from selected views. The viewscan be selected by algorithm, or can be selected by the user. FIG. 14shows a disparity map 1430 for the image. Given the configuration of theplenoptic otoscope, there is a one-to-one mapping between disparity anddepth. Therefore, the disparity map is a form of depth map, and viceversa. The disparity map is calculated according to:

{circumflex over (n)} _(p)(x,y)=arg max{corr(I ₁ , . . . ,I _(N))}  (1)

where {circumflex over (n)}_(p)(x,y) is the estimated disparity at pixel(x,y), I₁ . . . I_(N) are the translated images, and corr is acorrelation computation operator. The correlation can be calculatedeither globally or locally by using a sliding window. Different types ofcorrelation computations can be used, such as sum of absolute different,normalized cross correlation, multiplied eigenvalues of covariancematrix, phase correlation, etc. Further description is given in U.S.patent application Ser. No. 14/312,586, “Disparity estimation formultiview imaging systems,” which is incorporated by reference in itsentirety herein.

The disparity value assigned to the highest number of pixels in theimage is determined. In this example, that disparity value correspondsto a depth plane that is chosen 1340 as the reference plane. A histogram1440 of number of pixels at different disparities is shown in FIG. 14.The disparity of 0 has the largest number of pixels (bar 1442) and istherefore selected as the reference plane. The plenoptic data is used torefocus 1350 the image at the reference plane, as depicted by themultiviews 1450.

In this example, a synthetic aperture image 1460 is created 1360 byaveraging shifted (i.e., disparity-corrected) multiview images.Different number of views can be used in the averaging process to renderthe output image with different synthetic apertures. In this example,the synthesized image I_(s) is computed as a weighted average of theviews, according to:

I _(S)=(Σw _(i) V _(i)′)/(Σw _(i))  (2)

V_(i)′ is the ith view after shifting to account for disparity. Theimage shift could be done in the spatial domain or in the frequencydomain. w_(i) is a weighting factor, for example to compensate fornon-uniformity such as due to vignetting. The summation is over thenumber of views used to construct the synthesized image. FIG. 14 showstwo synthesized images 1460 a and 1460 b.

FIG. 15 a shows a synthesized image using a small aperture. FIG. 15 bshows the synthesized image using a large aperture. In the largeaperture example, all the views were included in the averaging process.It is also possible to select views to be included in the renderingaccording to an algorithm. Both images in FIGS. 15 a and 15 b are for ascene with hair occlusions. FIG. 15 c shows a reference image forcomparison. This reference image is without hair occlusions. Themultiviews of FIG. 15 b significantly reduce the amount of hairocclusion compared to the smaller aperture of FIG. 15 a.

Spectral responses of tissue or the TM can be measured by using spectralnarrow- or wide bandpass filters in the plenoptic otoscope. With suchspectral measurements, a characterization of the properties of the TM,such as translucency or coloration, can be obtained in conjunction withdepth measurements. Spectral measurements can be obtained for selectedlocations in the scene, e.g. on the TM. When choosing a near infrared(NIR) filter, longer wavelengths are penetrating the object at deeperlayers, making it, e.g. possible to obtain characterization of objectsbehind semi-translucent objects (e.g., behind the TM).

Spectral measurements of the ear canal can be obtained when inserting aspectral filter array into the lightfield otoscope. Examples aredescribed above with respect to FIGS. 5-6. The lightfield data obtainedfrom the sensor contain three-dimensional location as well as wavelengthspecific information. The spectral filters can be chosen such that theycapture spectral measurements of the ear canal with wide-band filters asused in conventional color imaging, and with narrow-band filters thatcapture certain specific information that is typically invisible to thehuman eye, e.g. certain type of amber coloration. When NIR filters areused, objects behind translucent object layers can be imaged, since NIRlight travels deeper through tissue before being reflected. The userwill obtain information of spectral reflectance of objects in the earcanal as well as a characterization of translucency of objects.

FIGS. 16 a-d show spectral images captured by a plenoptic otoscope. Thisexample uses four different wavebands: red, green, blue and amber. Theimages are obtained using a lightfield otoscope that has a filter modulewith four filters multiplexed in the aperture plane. FIGS. 16 a-d arethe red, green, blue and amber images, respectively.

Referring again to Table 1 above, the three conditions of the ear shownin Table 1 are different and they can be distinguished from one anotherbased on one or more of the following features: color, position (e.g.,3D shape), and translucency. In order to make correct diagnosis of theear condition, a plenoptic otoscopic can be used to capture accurateinformation about color, three-dimensional shape and/or translucency ofan inside of an ear (e.g., a tympanic membrane in an ear canal). Thesespectral measurements, individually or together with depth,polarization, translucency and/or bulging estimation, might be input toa machine learning algorithm to classify different medical conditions.The trained machine may be used to aid or automate diagnosis.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the invention but merely asillustrating different examples and aspects of the invention. It shouldbe appreciated that the scope of the invention includes otherembodiments not discussed in detail above. Various other modifications,changes and variations which will be apparent to those skilled in theart may be made in the arrangement, operation and details of the methodand apparatus of the present invention disclosed herein withoutdeparting from the spirit and scope of the invention as defined in theappended claims. Therefore, the scope of the invention should bedetermined by the appended claims and their legal equivalents.

What is claimed is:
 1. A method for making a medical diagnosis of an earinterior, comprising: acquiring plenoptic data of the ear interior; anda computer system processing the plenoptic data to produce diagnosticdata based on enhanced imagery of the ear interior, the enhanced imageryderived from the plenoptic data and the diagnostic data used in medicaldiagnosis of the ear interior.
 2. The method of claim 1 furthercomprising: the computer system processing the diagnostic data to makethe medical diagnosis.
 3. The method of claim 1 wherein the diagnosticdata includes the enhanced imagery.
 4. The method of claim 1 wherein theenhanced imagery includes a depth map of the tympanic membrane.
 5. Themethod of claim 4 wherein the depth map has sub-millimeter resolution.6. The method of claim 4 wherein processing the plenoptic data toproduce the depth map of the ear interior comprises: processing theplenoptic data by transforming to a scale and depth space.
 7. The methodof claim 4 further comprising: the computer system displaying avisualization of the depth map of the tympanic membrane, thevisualization used by a human in medical diagnosis of the ear interior.8. The method of claim 4 wherein the diagnostic data includesthree-dimensional information about the tympanic membrane, thethree-dimensional information derived from the enhanced imagery.
 9. Themethod of claim 8 wherein the three-dimensional information includeswhether the tympanic member is bulging or retracting.
 10. The method ofclaim 1 wherein the enhanced imagery includes different spectral imagesof the ear interior.
 11. The method of claim 10 wherein the spectralimages include an amber or yellow image.
 12. The method of claim 1wherein the enhanced imagery includes images showing translucency of theear interior.
 13. The method of claim 1 wherein the enhanced imagery isproduced for a selected reference plane.
 14. The method of claim 13wherein the reference plane is selected based on features in the earinterior.
 15. The method of claim 1 wherein processing the plenopticdata comprises combining multiviews of the plenoptic data to produce theenhanced imagery.
 16. The method of claim 15 wherein processing theenhanced imagery reduces occlusions.
 17. The method of claim 1 whereinthe medical diagnosis is otitis media.
 18. A tangible computer readablemedium containing software instructions than, when executed on acomputer system, cause the computer system to perform the steps of:accessing plenoptic data of an ear interior; processing the plenopticdata to produce diagnostic data based on enhanced imagery of the earinterior, the enhanced imagery derived from the plenoptic data and thediagnostic data used in medical diagnosis of the ear interior.
 19. Thetangible computer readable medium of claim 18 wherein the enhancedimagery includes a depth map of the tympanic membrane.
 20. The tangiblecomputer readable medium of claim 18 wherein the computer system furtherperforms the step of: processing the diagnostic data to make the medicaldiagnosis.